Why Johnny Can’t Use Agents: Industry Aspirations vs. User Realities with AI Agents

A mixed-methods study of 102 commercial AI agents and 31 end-user participants that quantifies the gap between what the industry markets AI agents as being able to do and what real users can actually accomplish. The findings surface recurring usability failure patterns and call into question whether current agent UX is ready for the mainstream use cases being advertised.

Abstract

There is growing imprecision about what “AI agents” are, what they can do, and how effectively they can be used by their intended users. We pose two key research questions: (i) How does the tech industry conceive and market “AI agents”? (ii) What challenges do end-users face when attempting to use commercial AI agents for their advertised uses? We first performed a systematic review of marketed use cases for 102 commercial AI agents, finding that they fall into three umbrella categories: orchestration, creation, and insight. We then evaluated whether end-users could realize these marketed capabilities in practice: we conducted a usability assessment where N = 31 participants attempted representative tasks for each of these categories on two popular commercial AI agent tools: Operator and Manus. We found that users were generally impressed with these agents but faced significant usability challenges ranging from agent capabilities that were misaligned with user mental models to agents lacking the meta-cognitive abilities necessary for effective collaboration.